Various applications such as video surveillance and news broadcast require multiple video streams to be displayed simultaneously on a single terminal device. Such video streams may be encoded in a diversity of video encoding standards and are often processed by a central unit such as multipoint control unit (MCU) before being transmitted to an intended terminal device such as a display screen, mobile phone, or a television. For example, in a video conference session involving communication between two or more participants, the MCU processes encoded video streams from the participants, and returns a dedicated re-encoded video stream in a predefined video codec standard such as H.264, also known as Moving Pictures Experts Group-4 (MPEG-4) Part 10 or MPEG-4 Advanced Video Coding (AVC), to each participant.
In the H.264/AVC standard, a video includes a series of pictures (or frames), with each frame consisting of a two-dimensional array of pixels. The pixels are divided into macroblocks (a 16×16 array of pixels). Each macroblock has a macroblock number; in general, the macroblocks are numbered starting at the top-left of the frame, in increasing order from left-to-right and top-to-bottom. The macroblocks can be grouped into slices, and the slices can be grouped into slice groups. Macroblocks within a slice are arranged in ascending order by macroblock number. A slice can include any number of macroblocks, which may or may not be contiguous; that is, macroblocks in one slice may be interspersed among macroblocks of one or more other slices of other slice groups; however, macroblocks from slices in the same slice group are not interspersed with each other. H.264 has a feature referred to as flexible macroblock ordering (FMO) that allows macroblocks to be grouped into slices.
FMO is one of the error resiliency tools that can be used by a decoder to conceal errors if slices are lost or corrupted during transmission. Macroblocks in a missing or corrupted slice can be reconstructed by interpolating or extrapolating macroblock information from another slice. More specifically, a correctly received slice can be decoded, and the information in that slice can be used to derive information for another slice.
Another H.264 feature is generally referred to as arbitrary slice ordering (ASO). With ASO, slices can be transmitted in any order. For example, a slice may be sent as soon as it is ready, that is, a slice may be streamed to a decoder as soon as all of the macroblocks, which make up that slice are encoded. As a result, a slice from one slice group may be sent, followed by a slice from another slice group, followed by another slice from the first slice group, and so on. Yet another feature of the H.264/AVC standard allows the MCU to implement the FMO and the ASO in combination with each other to generate an encoded video stream.
Typically, the MCU encodes video streams from multiple participants either separately or as a combined, encoded single video stream. Although the single video stream saves on the MCU computing cost, user experiences a lag in his own video at the terminal device. Therefore, the MCU usually encodes video streams from each participant separately to optimize the user experience, which increases the MCU processing or computing cost. Therefore, there exists a need for a solution that improves the MCU performance to optimally handle multiple video streams encoded in different video codec standards.